Dynamic administration of event pools for relevant event and alert analysis during event storms

ABSTRACT

Dynamic administration of event pools for relevant event and alert analysis during event storms including receiving, by an events analyzer from an events queue, a plurality of events from one or more components of the distributed processing system, each event including an occurred time and a logged time; creating, by the event analyzer, an events pool; determining whether an arrival rate of the events from the components of the distributed processing system is greater than a predetermined threshold; if the arrival rate is greater than the predetermined threshold, assigning, by the events analyzer, a plurality of events to the events pool in dependence upon their occurred time; and if the arrival rate is not greater than the predetermined threshold, assigning, by the events analyzer, a plurality of events to the events pool in dependence upon their logged time.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of and claims priorityfrom U.S. patent application Ser. No. 12/961,687, filed on Dec. 7, 2010.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with Government support under Contract No.HR0011-07-9-0002 awarded by the Department of Defense. The Governmenthas certain rights in this invention.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The field of the invention is data processing, or, more specifically,methods, systems, and products for dynamic administration of event poolsfor relevant event and alert analysis during event storms in adistributed processing system.

2. Description of Related Art

The development of the EDVAC computer system of 1948 is often cited asthe beginning of the computer era. Since that time, computer systemshave evolved into extremely complicated devices. Today's computers aremuch more sophisticated than early systems such as the EDVAC. Computersystems typically include a combination of hardware and softwarecomponents, application programs, operating systems, processors, buses,memory, input/output devices, and so on. As advances in semiconductorprocessing and computer architecture push the performance of thecomputer higher and higher, more sophisticated computer software hasevolved to take advantage of the higher performance of the hardware,resulting in computer systems today that are much more powerful thanjust a few years ago.

Modern distributed processing systems for intensive computing may havemillions of devices with many processes running on each device all ofwhich are capable of error and status reporting for automated errorrecovery, reporting to a systems administrator, and for other reasons.In many cases, in the case of an error for example, the sheer number ofsuch error reports and status reports are so overwhelming that theycannot be handled in a meaningful manner. For example, a systemsadministrator receiving a hundred thousand error reports may beoverwhelmed by the sheer number of such reports and therefore in theaggregate those reports become more and more unhelpful and irrelevant.

SUMMARY OF THE INVENTION

Methods, systems, and computer program products are provided for dynamicadministration of event pools for relevant event and alert analysisduring event storms including receiving, by an events analyzer from anevents queue, a plurality of events from one or more components of thedistributed processing system, each event including an occurred time anda logged time; creating, by the event analyzer, an events pool;determining whether an arrival rate of the events from the components ofthe distributed processing system is greater than a predeterminedthreshold; if the arrival rate is greater than the predeterminedthreshold, assigning, by the events analyzer, a plurality of events tothe events pool in dependence upon their occurred time; and if thearrival rate is not greater than the predetermined threshold, assigning,by the events analyzer, a plurality of events to the events pool independence upon their logged time.

The foregoing and other objects, features and advantages of theinvention will be apparent from the following more particulardescriptions of exemplary embodiments of the invention as illustrated inthe accompanying drawings wherein like reference numbers generallyrepresent like parts of exemplary embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary system for relevant alert delivery withevent and alert suppression in a distributed processing system accordingto embodiments of the present invention.

FIG. 2 sets forth a block diagram of automated computing machinerycomprising an exemplary computer useful in relevant alert delivery withevent and alert suppression according to embodiments of the presentinvention.

FIG. 3 sets forth a block diagram of an exemplary system for relevantalert delivery with event and alert suppression in a distributedprocessing system according to embodiments of the present invention.

FIG. 4 sets forth a diagram illustrating assigning events to an eventpool according to embodiments of the present invention.

FIG. 5 sets forth a diagram illustrating assigning alerts to an alertpool according to embodiments of the present invention.

FIG. 6 sets forth a flow chart illustrating an example method ofrelevant alert delivery with event and alert suppression according toembodiments of the present invention.

FIG. 7 sets forth a flow chart illustrating an exemplary method ofadministering incident pools for event and alert analysis in adistributed processing system according to embodiments of the presentinvention.

FIG. 8 sets forth a flow chart illustrating an additional method ofadministering incident pools for event and alert analysis in adistributed processing system according to embodiments of the presentinvention.

FIG. 9 sets forth a flow chart illustrating an additional method ofadministering incident pools for event and alert analysis in adistributed processing system according to embodiments of the presentinvention.

FIG. 10 sets forth a flow chart illustrating an example method ofdynamic administration of event pools for relevant event and alertanalysis during event storms according to embodiments of the presentinvention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Exemplary methods, systems, and computer program products for dynamicadministration of event pools for relevant event and alert analysisduring event storms and relevant alert delivery in a distributedprocessing system according to embodiments of the present invention aredescribed with reference to the accompanying drawings, beginning withFIG. 1. FIG. 1 illustrates an exemplary system for administeringincident pools for event and alert analysis, dynamic administration ofevent pools during event storms, and relevant alert delivery in adistributed processing system according to embodiments of the presentinvention. A distributed processing system is typically implemented asmultiple autonomous or semi-autonomous computers that communicatethrough a computer network. In such example distributed processingsystems, the computers often interact with each other in order toachieve a common goal. A computer program that runs in such an exampledistributed system is typically called a distributed program, anddistributed programming is often used to describe the process of writingsuch programs.

In the example of FIG. 1, the distributed processing system (101) isimplemented as a parallel computer (100), non-volatile memory for thecomputer in the form of data storage device (118), an output device forthe computer in the form of printer (120), and an input/output devicefor the computer in the form of computer terminal (122). The parallelcomputer (100) in the example of FIG. 1 also includes a plurality ofcompute nodes (102). Each compute node is an automated computing devicecomposed of one or more computer processors, its own computer memory,and its own input/output functionality. The compute nodes (102) arecoupled for data communications by several independent datacommunications networks including a high speed Ethernet network (174), aJoint Test Action Group (‘JTAG’) network (104), a tree network (106)which is optimized for collective operations, and a torus network (108)which is optimized for point to point operations. Tree network (106) isa data communications network that includes data communications linksconnected to the compute nodes so as to organize the compute nodes as atree. Each data communications network is implemented with datacommunications links among the compute nodes (102). The datacommunications links provide data communications for parallel operationsamong the compute nodes of the parallel computer.

In addition to compute nodes, computer (100) includes input/output(‘I/O’) nodes (110, 114) coupled to compute nodes (102) through one ofthe data communications networks (174). The I/O nodes (110, 114) provideI/O services between compute nodes (102) and I/O devices (118, 120,122). I/O nodes (110, 114) are connected for data communications throughlocal area network (‘LAN’) (130). Computer (100) also includes a servicenode (116) coupled to the compute nodes through one of the networks(104). Service node (116) provides service common to pluralities ofcompute nodes, loading programs into the compute nodes, starting programexecution on the compute nodes, retrieving results of program operationson the computer nodes, and so on. Service node (116) runs a serviceapplication (124) and communicates with users (128) through a serviceapplication interface (126) that runs on computer terminal (122).

Many of the components of the distributed processing system of FIG. 1,that is the devices of the distributed processing system or processesrunning on the devices of the distributed processing system of FIG. 1are capable of some form of error or status reporting through events andmany of such components are also capable of receiving alerts in responseto one or more of such events. Often in distributed processing systemsuseful according to embodiments of the present invention hundreds ofthousands or millions of components may provide incidents, often in theform of events or receive alerts.

In incident is, is a generic term used in this specification to mean anidentification or notification of a particular occurrence on a componentof a distributed processing system such as events described below, arefined identification of an occurrence often based on events such as analert described below, or other notifications as will occur to those ofskill in the art.

Incidents are administered in pools for event and alert analysisaccording to embodiments of the present invention. A pool of incidentsis a collection of incidents organized by the time of either theiroccurrence, by the time they are logged in an incident queue, includedin the pool, or other time as will occur to those of skill in the art.

The service node (116) of FIG. 1 has installed upon it an event andalert analysis module (124) that includes at least two incidentanalyzers implemented as events analyzers and alerts analyzers capableof administering incident pools for event and alert analysis in adistributed processing system according to embodiments of the presentinvention. The event and alert analysis module (124) is implemented asautomated computing machinery capable of receiving a plurality ofincidents from one or more components of the distributed processingsystem; creating a pool of incidents, the pool having a predeterminedinitial period of time; assigning each received incident to the pool;assigning to each incident a predetermined minimum time for inclusion ina pool; extending for one or more of the incidents the predeterminedinitial period of time of the pool by a particular period of timeassigned to the incident; determining whether conditions have been metto close the pool; and if conditions have been met to close the pool,determining for each incident in the pool whether the incident has beenin the pool for its predetermined minimum time for inclusion in a pool;and if the incident has been in the pool for us predetermined minimumtime, including the incident in the closed pool; and if the incident hasnot been in the pool for its predetermined minimum time, evicting theincident from the closed pool and including the incident in a next pool.

The event and alert analysis module (124) is also capable of relevantalert delivery with event and alert suppression in a distributedprocessing system according to embodiments of the present invention. Theevent and alert analysis module (124) of FIG. 1 is implemented asautomated computing machinery capable of receiving in an event queue aplurality of events from one or more components of a distributedprocessing system; assigning by an event analyzer each received event toan events pool; wherein the event analyzer includes event analysis rulesincluding, event arrival rules, events pool operation rules, eventsuppression rules, and events pool closure rules; identifying by theevent analyzer in dependence upon the event arrival rules and the eventsassigned to the events pool one or more alerts; closing, by the eventanalyzer in dependence upon the events pool operation rules, the eventspool; determining, by the events analyzer in dependence upon the eventsuppression rules, whether to suppress one or more events in the closedevents pool; and identifying by the event analyzer in dependence uponthe events pool closure rules and any unsuppressed events assigned tothe events pool, one or more additional alerts; sending by the eventanalyzer to an alert analyzer all the alerts identified by the eventanalyzer; assigning by the alert analyzer the identified alerts to analerts pool; determining by the alert analyzer in dependence upon alertanalysis rules and the alerts in the alert pool whether to suppress anyalerts; and transmitting the unsuppressed alerts to one or morecomponents of the distributed processing system.

In some embodiments the unsuppressed alerts are transmitted to one ormore components of the distributed processing system. One such componentmay be a terminal (122) for display to a systems administrator. Othercomponents may include a component that generated an event, a componentfor error reporting, a component for automated error recovery or anyother component that will occur to those of skill in the art.

The event and alert module (124) of FIG. 1 allows the number ofincidents occurring such as events received and alerts produced at anygiven time to be less overwhelming to a systems administrator (128)attempting to identify a problem or occurrence in the distributedprocessing system. Administering incident pools for event and alertanalysis and relevant alert delivery with event and alert suppressionaccording to embodiments of the present invention provide alerts thatare more meaningful to a user in determining how to administer thefunctions and errors associated with a distributed processing system.

Certain occurrences in a distributed processing system of FIG. 1 cancause an events storm. An events storm as that term is used here is theresult of an occurrence in the distributed processing system that causesan overwhelming number of events reported by components of thedistributed processing system. Such occurrences that may cause an eventsstorm may include the loss of an entire circuit providing power to manycomponents of the data processing system, catastrophic failure of anumber of components of the distributed processing system and others aswill occur to those of skill in the art. Events storms are often calledreliability, availability, and serviceability ‘RAS’ storms.

The arrival rate of the events that constitutes such an overwhelmingevents storm may vary according to various distributed processingsystems. In some embodiments, an arrival rate threshold may beestablished that identifies an events storm. Typically, such a thresholdmay be an arrival rate that exceeds a monitor or event analyzer'sability handle the events in an appropriately timely fashion.

The event and alert analysis module of FIG. 1 also includes automatedcomputing machinery capable of dynamic administration of event pools forrelevant event and alert analysis during event storms by receiving, byan events analyzer from an events queue, a plurality of events from oneor more components of the distributed processing system, each eventincluding an occurred time and a logged time; creating by the eventanalyzer an events pool; determining whether an arrival rate of theevents from the components of the distributed processing system isgreater than a predetermined threshold; if the arrival rate is greaterthan the predetermined threshold, assigning, by the events analyzer, aplurality of events to the events pool in dependence upon their occurredtime; and if the arrival rate is not greater than the predeterminedthreshold, assigning, by the events analyzer, a plurality of events tothe events pool in dependence upon their logged time.

The arrangement of nodes, networks, and I/O devices making up theexemplary distributed processing system illustrated in FIG. 1 are forexplanation only, not for limitation of the present invention.Distributed data processing systems capable of administering incidentpools for event and alert analysis, dynamic administration of eventpools during event storms, and relevant alert delivery according toembodiments of the present invention may include additional nodes,networks, devices, and architectures, not shown in FIG. 1, as will occurto those of skill in the art. The parallel computer (100) in the exampleof FIG. 1 includes sixteen compute nodes (102); parallel computerscapable of relevant alert delivery according to embodiments of thepresent invention sometimes include thousands of compute nodes. Inaddition to Ethernet and JTAG, networks in such data processing systemsmay support many data communications protocols including for example TCP(Transmission Control Protocol), IP (Internet Protocol), and others aswill occur to those of skill in the art. Various embodiments of thepresent invention may be implemented on a variety of hardware platformsin addition to those illustrated in FIG. 1.

Administering incident pools for event and alert analysis and relevantalert delivery with event and alert suppression in accordance with thepresent invention is generally implemented with computers, that is, withautomated computing machinery. In the system of FIG. 1, for example, allthe service nodes, I/O nodes, compute nodes, of the parallel computerare implemented to some extent at least as computers. For furtherexplanation, therefore, FIG. 2 sets forth a block diagram of automatedcomputing machinery comprising an exemplary computer (152) useful inrelevant alert delivery according to embodiments of the presentinvention. The computer (152) of FIG. 2 includes at least one computerprocessor (156) or ‘CPU’ as well as random access memory (168) (‘RAM’)which is connected through a high speed memory bus (166) and bus adapter(158) to processor (156) and to other components of the computer (152)and through an expansion bus to adapters for communications with othercomponents of a distributed processing system (101).

Stored in RAM (168) is an event and alert analysis module (124), amodule of automated computing machinery for relevant alert deliveryaccording to embodiments of the present invention. The event and alertanalysis module (124) includes two incident analyzers according toembodiments of the present invention. The incident analyzers include anevent analyzer (208) and an alert analyzer (218). The event analyzer ofFIG. 2 is a module of automated computing machinery capable ofidentifying alerts in dependence upon received events. That is, eventanalyzers typically receive events and produce alerts. In manyembodiments, a plurality of event analyzers are implemented in parallel.Often such event analyzers are assigned to a particular pool of eventsand may be focused on events from a particular component or caused by aparticular occurrence to produce a more concise set of alerts.

The alert analyzer of FIG. 2 is a module of automated computingmachinery capable of identifying alerts for transmission from events andother alerts, identifying additional alerts for transmission, andsuppressing unnecessary, irrelevant, or otherwise unwanted alertsidentified by the event analyzer. That is, alert analyzers typicallyreceive alerts and events and produce or forward alerts in dependenceupon those alerts and events. In many embodiments, a plurality of alertanalyzers are implemented in parallel. Often such alert analyzers areassigned to a particular pool of alerts and may be focused on alertswith particular attributes to produce a more concise set of alerts.

The event analyzer (208) and the alert analyzer are each capable ofadministering incident pools for event and alert analysis in adistributed processing system according to embodiments of the presentinvention. The event analyzer administers pools of events according toembodiments of the present invention by receiving, by the event analyzerfrom an event queue, a plurality of events from one or more componentsof the distributed processing system; creating, by the event analyzer, apool of events, the pool having a predetermined initial period of time;assigning, by the event analyzer, each received event to the pool;assigning, by the event analyzer, to each event a predetermined minimumtime for inclusion in a pool; extending, by the event analyzer, for oneor more of the events the predetermined initial period of time of thepool by a particular period of time assigned to the event; determining,by the event analyzer, whether conditions have been met to close thepool; and if conditions have been met to close the pool, determining foreach event in the pool whether the event has been in the pool for itspredetermined minimum time for inclusion in a pool; and if the event hasbeen in the pool for us predetermined minimum time, including the eventin the closed pool; and if the event has not been in the pool for itspredetermined minimum time, evicting the event from the closed pool andincluding the event in a next pool.

The event analyzer also dynamically administers event pools for relevantevent and alert analysis during event storms according to embodiments ofthe present invention by receiving, by an events analyzer from an eventsqueue, a plurality of events from one or more components of thedistributed processing system, each event including an occurred time anda logged time; creating by the event analyzer an events pool;determining whether an arrival rate of the events from the components ofthe distributed processing system is greater than a predeterminedthreshold; if the arrival rate is greater than the predeterminedthreshold, assigning, by the events analyzer, a plurality of events tothe events pool in dependence upon their occurred time; and if thearrival rate is not greater than the predetermined threshold, assigning,by the events analyzer, a plurality of events to the events pool independence upon their logged time.

The alert analyzer administers pools of alerts according to embodimentsof the present invention by receiving, by an alert analyzer from analert queue, a plurality of alerts from one or more components of thedistributed processing system; creating, by the alert analyzer, a poolof alerts, the pool having a predetermined initial period of time;assigning, by the alert analyzer, each received alert to the pool;assigning, by the alert analyzer, to each alert a predetermined minimumtime for inclusion in a pool; extending, by the alert analyzer, for oneor more of the alerts the predetermined initial period of time of thepool by a particular period of time assigned to the alert; determining,by the alert analyzer, whether conditions have been met to close thepool; and if conditions have been met to close the pool, determining foreach alert in the pool whether the alert has been in the pool for itspredetermined minimum time for inclusion in a pool; and if the alert hasbeen in the pool for us predetermined minimum time, including the alertin the closed pool; and if the alert has not been in the pool for itspredetermined minimum time, evicting the alert from the closed pool andincluding the alert in a next pool.

The event and alert analysis module (124) of FIG. 2 includes computerprogram instructions for receiving in an event queue a plurality ofevents from one or more components (for example, 100, 182, 181, 180, and170) of a distributed processing system (101); assigning by the eventanalyzer (208) each received event to an events pool; identifying by theevent analyzer (208) in dependence upon the event arrival rules and theevents assigned to the events pool one or more alerts; closing, by theevent analyzer (208) in dependence upon the events pool operation rules,the events pool; determining, by the events analyzer (208) in dependenceupon the event suppression rules, whether to suppress one or more eventsin the closed events pool; identifying by the event analyzer (208) independence upon the events pool closure rules and any unsuppressedevents assigned to the events pool, one or more additional alerts;sending by the event analyzer (208) to the alert analyzer (218) all thealerts identified by the event analyzer; assigning by the alert analyzer(218) the identified alerts to an alerts pool; determining by the alertanalyzer (218) in dependence upon alert analysis rules and the alerts inthe alert pool whether to suppress any alerts; and transmitting (420)the unsuppressed alerts to one or more components of the distributedprocessing system.

Also stored in RAM (168) is an operating system (154). Operating systemsuseful according to embodiments of the present invention include UNIX™,Linux™, Microsoft XP™, AIX™, IBM's i5/OS™, and others as will occur tothose of skill in the art. The operating system (154), event and alertanalysis module (124), the event analyzer (208), the alert analyzer(218) in the example of FIG. 2 are shown in RAM (168), but manycomponents of such software typically are stored in non-volatile memoryalso, such as, for example, on a disk drive (170).

The computer (152) of FIG. 2 includes disk drive adapter (172) coupledthrough expansion bus (160) and bus adapter (158) to processor (156) andother components of the computer (152). Disk drive adapter (172)connects non-volatile data storage to the computer (152) in the form ofdisk drive (170). Disk drive adapters useful in computers for accordingto embodiments of the present invention include Integrated DriveElectronics (‘IDE’) adapters, Small Computer System Interface (‘SCSI’)adapters, and others as will occur to those of skill in the art.Non-volatile computer memory also may be implemented for as an opticaldisk drive, electrically erasable programmable read-only memory(so-called ‘EEPROM’ or ‘Flash’ memory), RAM drives, and so on, as willoccur to those of skill in the art.

The example computer (152) of FIG. 2 includes one or more input/output(‘I/O’) adapters (178). I/O adapters implement user-orientedinput/output through, for example, software drivers and computerhardware for controlling output to display devices such as computerdisplay screens, as well as user input from user input devices (181)such as keyboards and mice. The example computer (152) of FIG. 2includes a video adapter (209), which is an example of an I/O adapterspecially designed for graphic output to a display device (180) such asa display screen or computer monitor. Video adapter (209) is connectedto processor (156) through a high speed video bus (164), bus adapter(158), and the front side bus (162), which is also a high speed bus.

The exemplary computer (152) of FIG. 2 includes a communications adapter(167) for data communications with other computers (182) and for datacommunications with a data communications network (100). Such datacommunications may be carried out serially through RS-232 connections,through external buses such as a Universal Serial Bus (‘USB’), throughdata communications data communications networks such as IP datacommunications networks, and in other ways as will occur to those ofskill in the art. Communications adapters implement the hardware levelof data communications through which one computer sends datacommunications to another computer, directly or through a datacommunications network. Examples of communications adapters useful forrelevant alert delivery according to embodiments of the presentinvention include modems for wired dial-up communications, Ethernet(IEEE 802.3) adapters for wired data communications networkcommunications, and 802.11 adapters for wireless data communicationsnetwork communications.

For further explanation, FIG. 3 sets forth a block diagram of anexemplary system for administering incident pools for event and alertanalysis, dynamic administration of event pools for relevant event andalert analysis during event storms, and relevant alert delivery in adistributed processing system (102) according to embodiments of thepresent invention. The system of FIG. 3 includes an event and alertanalysis module (124). The event and alert analysis module (124) of FIG.3 receives in an event queue (206) a plurality of events (202) from oneor more components of a distributed processing system (102). A componentof a distributed processing system according to embodiments of thepresent invention may be a device of the distributed processing systemor a process running on a device of the distributed processing. Suchcomponents are often capable of some form event transmission, often forerror or status reporting.

An event according to embodiments of the present invention is anotification of a particular occurrence in or on a component of thedistributed processing system. Such events are sent from the componentupon which the occurrence occurred or another reporting component to anevent and alert analysis module according to the present invention.Often events are notifications of errors occurring in a component of thedata processing system. Events are often implemented as messages eithersent through a data communications network or shared memory. Typicalevents for event and alert analysis according to embodiments of thepresent invention an occurred time, a logged time, an event type, anevent ID, a reporting component, and a source component, and otherattributes. An occurred time is the time at which the event occurred onthe component. A logged time is the time the event was included in theevent queue (206) and is typically inserted into the event by themonitor (204) in the example of FIG. 3. An event type is a generic typeof event such as for example, power error, link failure error, errorsrelated to not receiving messages or dropping packets and so on as willoccur to those of skill in the art. An event ID is a uniqueidentification of the event. A reporting component is an identificationof the component that reported the event. A source component is anidentification of the component upon which the event occurred. In manycases, but not all, the reporting component and source component are thesame component of the distributed processing system.

In the example of FIG. 3, the event and alert analysis module (124)includes a monitor (204) that receives events from components of thedistributed processing system and puts the received events (202) in theevent queue (206). The monitor (204) of FIG. 3 may receive events fromcomponents of the distributed processing system on their motion, mayperiodically poll one or more of the components of the distributedprocessing system, or receive events from components in other ways aswill occur to those of skill in the art.

They system of FIG. 3 includes an event analyzer (208). The eventanalyzer (208) of FIG. 3 is a module of automated computing machinerycapable of identifying alerts in dependence upon received events. Thatis, event analyzers typically receive events and produce alerts. In manyembodiments, a plurality of event analyzers are implemented in parallel.Often event analyzers are assigned to a particular pool of events andmay be focused on events from a particular component or caused by aparticular occurrence to produce a more concise set of alerts.

The event analyzer (208) of FIG. 3 assigns each received event (202) toan events pool (212). An events pool (212) is a collection of eventsorganized by the time of either their occurrence, by the time they arelogged in the event queue, included in the events pool, or other time aswill occur to those of skill in the art. That is, event pools are acollection of events organized by time. Such events pools often providethe ability to analyze a group of time related events identify alerts independence upon them. Often such event pools are useful in identifyingfewer and more relevant alerts in dependence upon multiple relatedevents.

The events pool (212) is administered by the event analyzer (208)according to embodiments of the present invention. The event analyzeradministers pools of events according to embodiments of the presentinvention by receiving, by the event analyzer from the event queue, aplurality of events from one or more components of the distributedprocessing system; creating, by the event analyzer, a pool of events,the pool having a predetermined initial period of time; assigning, bythe event analyzer, each received event to the pool; assigning, by theevent analyzer, to each event a predetermined minimum time for inclusionin a pool; extending, by the event analyzer, for one or more of theevents the predetermined initial period of time of the pool by aparticular period of time assigned to the event; determining, by theevent analyzer, whether conditions have been met to close the pool; andif conditions have been met to close the pool, determining for eachevent in the pool whether the event has been in the pool for itspredetermined minimum time for inclusion in a pool; and if the event hasbeen in the pool for us predetermined minimum time, including the eventin the closed pool; and if the event has not been in the pool for itspredetermined minimum time, evicting the event from the closed pool andincluding the event in a next pool.

As mentioned an events pool according to the method of FIG. 3 has apredetermined initial period of time and in the example of FIG. 3assigning by the event analyzer each received event to an events poolincludes extending for each event assigned to the events pool thepredetermined initial period of time by a particular period of timeassigned to the event. In this manner, the pool is extended with eachreceived event until a collection of events that may be usefully used toidentify alerts is assigned to the events pool.

As mentioned above, in some embodiments of the present invention, morethan one event analyzer may operate in parallel. As such, each eventanalyzer may maintain one or more event pools for relevant alertdelivery according to embodiments of the present invention. Assigning bythe event analyzer the events to an events pool may therefore includeselecting only events from one or more particular components. In suchembodiments, particular components may be selected for a particularevents pool to provide events associated with a particular period oftime from a particular set of one or more components.

Assigning by the event analyzer the events to an events pool may also becarried out by selecting only events of a particular event type. In suchembodiments, particular events may be selected for a particular eventspool to provide events associated with a particular period of time froma particular set of event types.

As mentioned above, certain occurrences in a distributed processingsystem of FIG. 1 can cause an events storm. The event analyzer (208) ofFIG. 3 therefore also dynamically administers event pools for relevantevent and alert analysis during event storms according to embodiments ofthe present invention by receiving, from an events queue, a plurality ofevents from one or more components of the distributed processing system,each event including an occurred time and a logged time; creating anevents pool; determining whether an arrival rate of the events from thecomponents of the distributed processing system is greater than apredetermined threshold; if the arrival rate is greater than thepredetermined threshold, assigning a plurality of events to the eventspool in dependence upon their occurred time; and if the arrival rate isnot greater than the predetermined threshold, assigning a plurality ofevents to the events pool in dependence upon their logged time.

Event analyzer (208) in the example of FIG. 3 identifies in dependenceupon the event analysis rules (210) and the events assigned to theevents pool one or more alerts (214). Event analyses rules (210) are acollection of predetermined rules for meaningfully parsing receivedevents to identify relevant alerts in dependence upon the events.

The event analysis rules (210) of FIG. 3 include event arrival rules(230), events pool operation rules (232), event suppression rules (234),and events pool closure rules (236). Event arrival rules (230) areconfigurable predetermined rules for identifying alerts in dependenceupon events in real time as those events are assigned to the eventspool. That is, event arrival rules (230) identify alerts in dependenceupon events before closing the events pool. Such rules are typicallypredetermined to identify particular alerts in dependence uponattributes of those events. Event arrival rules may for example dictateidentifying a particular predetermined alert for transmission to asystems administrator in dependence upon a particular event type orcomponent type for the event or other attribute of that event. Suchrules are flexible and may be tailored to a particular distributedcomputing system and its functions.

An alert according to embodiments of the present invention is refinedidentification of an occurrence—such and an error—based upon more thanone event and therefore provides an identification of the occurrence inthe context of its operation in the distributed processing system. Oftenan alert may be a notification of a particular error type of occurrencethat is identified in dependence upon the plurality of events receivedfrom one or more components of the data processing system, such as, forexample, a link failure among a plurality of devices each of which areproducing many events based upon the single link failure, or a powerfailure provoking thousands of events, and so on.

Alerts are often implemented as messages to be sent through a datacommunications network or shared memory. Typical alerts according toembodiments of the present invention have attributes attached to thembased upon the attributes of the events received from which they areidentified.

Events pool operation rules (232) are configurable predetermined rulesfor controlling the operations of the events pool. Such rules includesrules identifying the initial predetermined period of time for eachevents pool, rules dictating the length of time extended to the poolupon the assignment of each new event to the pool, rules dictating theminimum time an event must be in a pool before that event is included ina collection of events when the pool is closed, rules governing theclosing of an events pool, and others as will occur to those of skill inthe art. Such rules are flexible and may be tailored to a particulardistributed computing system and its functions.

Event suppression rules (234) are configurable predetermined rules forsuppressing one or more events in a closed pool of events used inidentifying alerts. That is, often events in the closed events pool maybe duplicate events, redundant events, or otherwise unnecessary orunhelpful events in identifying alerts. Such suppression rules aretypically predetermined to delete, drop, or otherwise ignore thosesuppressed events. Event suppression rules may for example dictate thatmore than a threshold number of events of a particular event type orcomponent type are to be suppressed. Such rules are also flexible andmay be tailored to a particular distributed computing system and itsfunctions.

Events pool closure rules (236) are configurable predetermined rules foridentifying alerts in dependence upon unsuppressed events in the closedevents pool and alerts identified by the event arrival rules. That is,events pool closure rules identify new alerts in dependence upon one ormore or even all of the unsuppressed events in the closed events pool.The events pool closure rules also identify alerts in dependence uponthe alerts identified by the event arrival rules (230) or a combinationof the alerts identified by the event arrival rules (230) and one ormore of the unsuppressed events in the closed events pool.

Event analyzer (208) in the example of FIG. 3 sends all the alerts (214)identified by the event analyzer (208) to an alert analyzer (218). Thealert analyzer of FIG. 3 is a module of automated computing machinerycapable of identifying alerts for transmission from events and otheralerts, identifying additional alerts for transmission, and suppressingunnecessary, irrelevant, or otherwise unwanted or unhelpful alertsidentified by the event analyzer. That is, alert analyzers typicallyreceive alerts and events and produce or forward alerts in dependenceupon those alerts and events. In many embodiments, a plurality of alertanalyzers are implemented in parallel. The alerts (216) in the exampleof FIG. 3 are sent from event analyzer (208) to an alert analyzer (218)through an alerts queue (216).

The alert analyzer (218) of FIG. 3 assigns each of the identified alerts(214) to an alerts pool (224). An alerts pool (224) is a collection ofalerts organized by the time of one or more the events causing the alertto be identified, the time the alert is identified, or other time aswill occur to those of skill in the art. That is, alerts pools are acollection of alerts organized by time. Such alerts pools often providethe ability to analyze a groups alerts identified and included in thealerts pool according to some time. Often such alerts pools are usefulin identifying fewer and more relevant alerts in dependence uponmultiple related events and multiple related alerts.

The alert analyzer administers pools of alerts according to embodimentsof the present invention by receiving, by an alert analyzer from analert queue, a plurality of alerts from one or more components of thedistributed processing system; creating, by the alert analyzer, a poolof alerts, the pool having a predetermined initial period of time;assigning, by the alert analyzer, each received alert to the pool;assigning, by the alert analyzer, to each alert a predetermined minimumtime for inclusion in a pool; extending, by the alert analyzer, for oneor more of the alerts the predetermined initial period of time of thepool by a particular period of time assigned to the alert; determining,by the alert analyzer, whether conditions have been met to close thepool; and if conditions have been met to close the pool, determining foreach alert in the pool whether the alert has been in the pool for itspredetermined minimum time for inclusion in a pool; and if the alert hasbeen in the pool for us predetermined minimum time, including the alertin the closed pool; and if the alert has not been in the pool for itspredetermined minimum time, evicting the alert from the closed pool andincluding the alert in a next pool.

The alert analyzer may assign the identified alerts to an alerts pool(224) in dependence upon attributes of the alerts or attributes of theevents from which those alerts were identified. For example, the alertanalyzer of FIG. 3 may assign alerts to the alerts pool (224) byselecting alerts generated from events from one or more particularcomponents, alerts associated with a particular alert type and so on aswill occur to those of skill in the art.

The alert analyzer (218) of FIG. 3 determines in dependence upon alertanalysis rules (222) and the alerts in the alert pool whether tosuppress any alerts. Suppressing an alert is typically carried out bydropping the alert, deleting the alert or otherwise ignoring or nottransmitting the suppressed alert to a component of the distributedprocessing system.

Alert analyses rules (222) are a collection of rules for suppressing oneor more alerts to provide a more relevant set of alerts for transmissionto a component of the distributed processing system, such as forexample, for display to a systems administrator and to identifyadditional alerts for transmission to one or more components of thedistributed processing system. Alert analysis rules for example maydictate that duplicate alerts are to be suppressed, alerts of aparticular type for transmission to a particular component are to besuppressed, alerts of a particular type be transmitted to a particularcomponent are to be suppressed and so on as will occur to those of skillin the art. Such alerts may be more meaningful to a component of thedistributed processing system for automated error recovery or for asystems administrator who may otherwise be less informed by a number ofraw unanalyzed alerts.

The alert analyzer (218) of FIG. 3 also has access to the events queue(206). The alert analyzer (218) of FIG. 3 in dependence upon the alertanalysis rules may, in some embodiments select events from the eventsqueue and determine whether to suppress any alerts in dependence uponthe selected events. That is, alert analysis rules may also take intoaccount events and their attributes for suppressing alerts and foridentifying additional alerts for transmission to one or morecomponents. Such events may be related to the alerts in the alerts poolor independent from such alerts.

The alert analyzer (218) of FIG. 3 transmits the unsuppressed alerts toone or more components of the distributed processing system. The alertanalyzer may transmit the unsuppressed alerts to one or more componentsof the distributed processing system by sending the alert as a messageacross a data communications network, through shared memory, or in otherways as will occur to those of skill in the art. In the example of FIG.3, the unsuppressed alerts (220) are transmitted to a terminal (122) fordisplay to a systems administrator (128).

The alert analyzer (218) of FIG. 3 is also capable of identifying independence upon alert analysis rules (222), the alerts in the alert pool(224), and selected events (206) one or more additional alerts andtransmitting the one or more components of the distributed processingsystem. The additional alerts may include one or more alerts notidentified by the event analyzer. Such additional alerts may provideadditional information to a component of the distributed processingsystem of a systems administrator.

As mentioned above, relevant alert delivery according to the presentinvention includes assigning events to an event pool and those pools areadministered according to embodiments of the present invention. Forfurther explanation, FIG. 4 sets forth a diagram illustrating assigningevents to an event pool according to embodiments of the presentinvention. An events pool (212) is a collection of events organized bythe time of either their occurrence, by the time they are logged in theevent queue, included in the events pool, or other time as will occur tothose of skill in the art. That is, event pools are a collection ofevents organized by time. Such events pools often provide the ability toanalyze a group of time related events and to identify alerts independence upon them. Often such event pools are useful in identifyingfewer and more relevant alerts in dependence upon multiple relatedevents.

Events pools according to embodiments of the present invention aretypically operated according to events pool operation rules which arethemselves often included in event analysis rules. Such events pooloperation rules are configurable predetermined rules for controlling theoperations of the events pool. Such rules includes rules identifying theinitial predetermined period of time for each events pool, rulesdictating the length of time extended to the pool upon the assignment ofeach new event to the pool, rules dictating the minimum time an eventmust be in a pool before that event is included in a collection ofevents when the pool is closed, rules governing the closing of an eventspool, and others as will occur to those of skill in the art. Such rulesare flexible and may be tailored to a particular distributed computingsystem and its functions.

Events are often assigned to an events pool according to their loggedtime. That is, events are typically inserted into the events pool in theorder that they are received in the event queue. In the example of FIG.4, the timing of the events pool (212) is initiated when the first event‘Event 0’ (400) is assigned to the events pool (212) at time t₀. Theevents pool of FIG. 4 is initiated for a predetermined initial period oftime from t₁ to t_(f). That is, upon receiving the first event ‘Event 0’(400) the events pool of FIG. 4 has a predetermined initial period oftime beginning at t₁ and ending at t_(f). The predetermined initialperiod of time may be configured in dependence upon a number of factorsas will occur to those of skill in the art such as, the number ofcomponents in the distributed processing system, the frequency ofreceiving events, the types of events typically received and so on aswill occur to those of skill in the art.

In the example FIG. 4, the initial period of time is extended for eachnew event assigned to the events pool during the predetermined initialperiod from t₁ to t_(f) by a particular period of time assigned to theevent. In the example of FIG. 4 upon assigning ‘Event 1’ (402) to theevents pool (212) the predetermined initial period of time t₀-t_(f) isextended by ‘Extension 1’ (406) having a time of e1 thereby creating anew time for closing the events pool (212) at t_(f+e1) if no otherevents are assigned to the pool before t_(f+e1). Similarly, in theexample of FIG. 4 upon assigning ‘Event 2’ (404) to the events poolhaving a time of e2, the now extended period of time from t₀-t_(f+e1) isextended again by extension 2 (406) thereby establishing a new time forclosing the pool at time t_(f+e1+e2) if no other events are assigned tothe pool before t_(f+e1+e2) or before some maximum time for the eventspool has expired. In this manner, the event pool is extended with eachreceived event until a collection of events that may be usefully used toidentify alerts is assigned to the events pool.

In typical embodiments of the present invention, event pools may have amaximum duration that can no longer be extended. In such cases, arequirement may exist that an event that has not resided in the eventpool for a threshold period of time be moved to a next events pool. Insome embodiments, the attributes of such an event that is moved to thenext events pool are used for relevant alert delivery according toembodiments of the present invention with the initial events pool and inother embodiments, the attributes of such an event are used for relevantalert delivery with the next events pool to which that event is moved.

In the example of FIG. 4, when conditions are met to close the pool anevents analyzer determines for each event (400, 402, 404) in the pool(212) whether the event has been in the pool for its predeterminedminimum time for inclusion in a pool. If the event has been in the poolfor its predetermined minimum time, the event is included in the closedpool for event analysis for relevant alert delivery according toembodiments of the present invention. If the event has not been in thepool for its predetermined minimum time, the event is evicted from theclosed pool and included a next pool for event analysis for relevantalert delivery according to embodiments of the present invention.

In many embodiments, a plurality of events pools may be used in paralleland one or more of such events pools are assigned to a particular eventsanalyzer. In such embodiments, events analyzers may be directed toevents in events pools having particular attributes.

As mentioned above, events are often assigned to an events poolaccording to their logged time. That is, events are typically insertedinto the events pool in the order that they are received in the eventqueue. However, during events storms it is often preferable toadminister events in the events pool in dependence upon their occurredtime rather than their logged time. In the example of FIG. 3, during anevent storms an event analyzer determines that an arrival rate of theevents from the components of the distributed processing system isgreater than a predetermined threshold and assigns the events to theevents pool upon their occurred time. That is, during an events storm t₀for ‘Event 0’ (400) is the occurred time of the event rather than thelogged time of the event, similarly t₁ for ‘Event 1’ (402) is theoccurred time of ‘Event 1’ and t₂ for ‘Event 2’ (404) is the occurredtime of ‘Event 2.’ Administration of the pool may operate as describedabove only with occurred times for the events during events stormsrather than logged times for those events.

Assigning events to an events pool according to their occurred timerather than their logged time may in some embodiments also affect themanner in which the timing of the events pool is administered. Whenassigning events to the events pool according to their logged time atimer may be used to administer the predetermined initial period of timeof the pool and extensions. During events storms, such a timer is lessuseful as the events are often received out of the order in which theyare created. As such, when assigning events to events pools according tologged times the timing of the pool is administered formulaically bycalculating the predetermined initial period of time from occurred timeof the first event assigned to the pool and then providing extensions oftime formulaically as more events are assigned to the pool.

As mentioned above, relevant alert delivery according to the presentinvention also includes assigning alerts to an alerts pool. For furtherexplanation, FIG. 5 sets forth a diagram illustrating assigning alertsto an alert pool according to embodiments of the present invention. Thealerts pool (224) of FIG. 5 operates in a manner similar to the eventspool of FIG. 4. That is, the alerts pool according to the example ofFIG. 5 includes alerts and the timing of the alerts pool begins with thefirst alert ‘Alert 0’ (500) at time t₀ and is configured to have apredetermined initial period of time t₀-t_(f). In the example of FIG. 5,the initial period of time is extended for each new alert assigned tothe alerts pool in the predetermined initial period from t₁ to t_(f) bya particular period of time assigned to the alert. In the example ofFIG. 5, upon assigning ‘Alert 1’ (502) to the alerts pool (224) thepredetermined initial period of time t₀-t_(f) is extended by ‘Extension1’ (506) having a time of e1 thereby creating a new time for closing thealerts pool (224) at t_(f+e1) if no other alerts are assigned to thepool before t_(f+e1). Similarly, in the example of FIG. 4 upon assigning‘Alert 2’ (504) to the alerts pool having a time of e2, the now extendedperiod of time from t₀-t_(f+e1) is extended again by ‘Extension 2’ (406)thereby establishing a new time for closing the pool at time t_(f+e1+e2)if no other alerts are assigned to the pool before t_(f+e1+e2) or beforesome maximum time for the alerts pool has expired.

In typical embodiments of the present invention, alerts pools may have amaximum duration that can no longer be extended. In such cases, arequirement may exist that an alert that has not resided in the alertpool for a threshold period of time be moved to a next alerts pool. Insome embodiments, the attributes of such an alert that is moved to thenext alerts pool are used for relevant alert delivery according toembodiments of the present invention with the initial alerts pool and inother embodiments, the attributes of such an alert are used for relevantalert delivery with the next alerts pool to which that alert is moved.

In the example of FIG. 5, when conditions are met to close the pool analerts analyzer determines for each alert (500, 502, 504) in the pool(224) whether the alert has been in the pool for its predeterminedminimum time for inclusion in a pool. If the alert has been in the poolfor its predetermined minimum time, the alert is included in the closedpool for alert analysis for relevant alert delivery according toembodiments of the present invention. If the alert has not been in thepool for its predetermined minimum time, the alert is evicted from theclosed pool and included a next pool for alert analysis for relevantalert delivery according to embodiments of the present invention.

In many embodiments, a plurality of alerts pools may be used in paralleland one or more of such alerts pools are assigned to a particular alertsanalyzer. In such embodiments, alerts analyzers may be directed toalerts in alerts pools having particular attributes.

For further explanation, FIG. 6 sets forth a flow chart illustrating anexample method of relevant alert delivery with event and alertsuppression in a distributed processing system according to embodimentsof the present invention. The method of FIG. 6 includes receiving (402)in an event queue a plurality of events (202) from one or morecomponents of a distributed processing system. Events useful in relevantalert delivery with event and alert suppression according to embodimentsof the present invention may include an occurred time, a logged time, anevent type, an event ID, a reporting component, and a source component.

Receiving (402) in an event queue a plurality of events (202) from oneor more components of a distributed processing system may be carried outby receiving an event initiated by one or more components of the dataprocessing system and storing the event in the event queue according tothe time in which the event occurred or according to the time the eventwas received. Receiving (402) in an event queue a plurality of events(202) from one or more components of a distributed processing systemalso may be carried out by polling a component for status and receivingin response an event and storing the event in the event queue accordingto the time in which the event occurred or according to the time theevent was received.

The method of FIG. 6 also includes assigning (404) by an event analyzereach received event to an events pool (212). In some embodiments of thepresent invention, assigning (404) by an event analyzer each receivedevent (202) to an events pool (212) may be carried out by assigningevents to the event pool according to the logged time. Assigning (404)by an event analyzer each received event (202) to an events pool (212)may also be carried out in dependence upon attributes of the event. Suchattributes may include an identification or type of the component uponwhich an occurrence occurred to create the event, the reportingcomponent of the event, the event ID, the event type, and so on as willoccur to those of skill in the art.

An events pool according to the method of FIG. 6 includes eventsoccurring during a predetermined initial period of time and in theexample of FIG. 6 assigning (404) by the event analyzer each receivedevent to an events pool includes extending (426) for each event assignedto the events pool the predetermined initial period of time by aparticular period of time assigned to the event.

The event analyzer includes event analysis rules (210) including, eventarrival rules, events pool operation rules, event suppression rules, andevents pool closure rules. Event arrival rules are configurablepredetermined rules for identifying alerts in dependence upon events inreal time as those events are assigned to the events pool. That is,event arrival rules identify alerts in dependence upon events beforeclosing the events pool. Such rules are flexible and may be tailored toa particular distributed computing system and its functions.

An alert according to embodiments of the present invention is refinedidentification of an occurrence—such and an error—based upon more thanone event and therefore provides an identification of the occurrence inthe context of its operation in the distributed processing system. Oftenan alert may be a notification of a particular error type of occurrencethat is identified in dependence upon the plurality of events receivedfrom one or more components of the data processing system, such as, forexample, a link failure among a plurality of devices each of which areproducing many events based upon the single link failure, or a powerfailure provoking thousands of events, and so on.

Alerts are often implemented as messages to be sent through a datacommunications network or shared memory. Typical alerts according toembodiments of the present invention have attributes attached to thembased upon the attributes of the events received from which they areidentified.

Events pool operation rules are configurable predetermined rules forcontrolling the operations of the events pool. Such rules includes rulesidentifying the initial predetermined period of time for each eventspool, rules dictating the length of time extended to the pool upon theassignment of each new event to the pool, rules dictating the minimumtime an event must be in a pool before that event is included in acollection of events when the pool is closed, rules governing theclosing of an events pool, and others as will occur to those of skill inthe art. Such rules are flexible and may be tailored to a particulardistributed computing system and its functions.

Event suppression rules are configurable predetermined rules forsuppressing one or more events in a closed pool of events used inidentifying alerts. That is, often events in the closed events pool maybe duplicate events, redundant events, or otherwise unnecessary orunhelpful events in identifying alerts. Such suppression rules aretypically predetermined to delete, drop, or otherwise ignore thosesuppressed events. Event suppression rules may for example dictate thatmore than a threshold number of events of a particular event type orcomponent type are to be suppressed. Such rules are also flexible andmay be tailored to a particular distributed computing system and itsfunctions.

Events pool closure rules are configurable predetermined rules foridentifying alerts in dependence upon unsuppressed events in the closedevents pool and alerts identified by the event arrival rules. That is,events pool closure rules identify new alerts in dependence upon one ormore or even all of the unsuppressed events in the closed events pool.The events pool closure rules also identify alerts in dependence uponthe alerts identified by the event arrival rules or a combination of thealerts identified by the event arrival rules and one or more of theunsuppressed events in the closed events pool.

The method of FIG. 6 also includes identifying (410) by the eventanalyzer in dependence upon the event arrival rules and the eventsassigned to the events pool one or more alerts (214). Identifying (410)by the event analyzer in dependence upon the event arrival rules and theevents assigned to the events pool one or more alerts (214) may becarried out by identifying alerts in dependence upon one or moreattributes of the events as that event is assigned to the events pool.Identifying (410) by the event analyzer in dependence upon the eventarrival rules and the events assigned to the events pool one or morealerts (214) may be carried by comparing the attributes of the events tothe event arrival rules and identifying as a result of the comparisonone or more alerts. Such attributes may include the type of componentfrom which the event was received, the type of component creating theevent, the identification of the component creating the event, the timethe event was created or received, an error reported in the event, andmany others as will occur to those of skill in the art.

The method of FIG. 6 also includes closing (412), by the event analyzerin dependence upon the events pool operation rules, the events pool(212). Closing (412), by the event analyzer in dependence upon theevents pool operation rules, the events pool (212) may be carried out bydetermining that conditions dictated by the events pool operation ruleshave been met to stop assigning new events to the events pool andidentifying in dependence upon those events pool operation rules theparticular events that are included in the closed pool of events.

Closing the events pool may be carried out by determining that theinitial period of time for the events pool and any particular periods oftime for events received in the events pool extended to the initialperiod of time have expired. In such cases, if no new events arereceived prior to the expiration of the initial period of time for theevents pool and any particular periods of time for events received inthe events pool extended to the initial period of time the pool isclosed.

Closing the events pool may also be carried out by determining that amaximum duration for the events pool has expired. In such cases,regardless of the number of new events being received after a maximumduration for the events pool has expired the pool is closed. In suchembodiments, a maximum duration for the events pool prevents the eventspool from including more events than are useful for relevant alertdelivery according to embodiments of the present invention.

The method of FIG. 6 also includes determining (414), by the eventsanalyzer in dependence upon the event suppression rules, whether tosuppress one or more events in the closed events pool (212). Determining(414), by the events analyzer in dependence upon the event suppressionrules, whether to suppress one or more events in the closed events pool(212) may be carried out by determining in dependence upon theattributes of one or more events in the closed pool whether to delete,drop, or otherwise ignore one or more of the events in the closed pool.

The method of FIG. 6 includes identifying (416) by the event analyzer independence upon the events pool closure rules and any unsuppressedevents assigned to the events pool, one or more additional alerts (417).Identifying (416) by the event analyzer in dependence upon the eventspool closure rules and any unsuppressed events assigned to the eventspool, one or more additional alerts (417) may be carried out byidentifying alerts in dependence upon one or more attributes of theevents as that event is assigned to the events pool. Identifying (416)by the event analyzer in dependence upon the events pool closure rulesand any unsuppressed events assigned to the events pool, one or moreadditional alerts (417) may be carried out by selecting the unsuppressedevents for the events pool, comparing the attributes of the unsuppressedevents of the events pool to the pool closure rules, and identifying asa result of the comparison one or more additional alerts. Suchattributes may include the type of component from which one or more ofthe unsuppressed events are received, the type of components creatingthe unsuppressed events, the identification of the component creatingthe unsuppressed events, the time the events were created or received,one or more errors reported by the events event, the number of events inthe pool, and many others as will occur to those of skill in the art.

The method of FIG. 6 includes sending (418) by the event analyzer to analert analyzer all the alerts identified by the event analyzer. Sending(418) by the event analyzer to an alert analyzer all the alerts (214)identified by the event analyzer may be carried out by sending a messagecontaining the alerts from the event analyzer to the alert analyzer.Such a message may be sent from the event analyzer to the alert analyzeracross a network, through shared memory, or in other ways as will occurto those of skill in the art.

The method of FIG. 6 includes assigning (420) by the alert analyzer theidentified alerts to an alerts pool (224). An alerts pool according tothe method of FIG. 6 has a predetermined initial period of time and inthe example of FIG. 6 assigning (420) by the alert analyzer theidentified alerts to an alerts pool (224) includes extending for eachalert assigned to the alerts pool the predetermined initial period oftime by a particular period of time assigned to the alert. Assigning(420) by the alert analyzer the identified alerts to an alerts pool(224) also may be carried out in dependence upon attributes of thealerts. Such attributes may include an identification or type of thecomponent upon which an occurrence occurred to create the event that wasused to identify the alert, the alert ID, the alert type, and so on aswill occur to those of skill in the art.

The method of FIG. 6 includes determining (422) by the alert analyzer independence upon alert analysis rules (222) and the alerts in the alertpool whether to suppress any alerts. Determining (422) by the alertanalyzer in dependence upon alert analysis rules (222) and the alerts inthe alert pool whether to suppress any alerts may be carried out independence upon one or more attributes of the alerts. Such attributesmay include an identification or type of the component upon which anoccurrence occurred to create the event that was used to identify thealert, the alert ID, the alert type, and so on as will occur to those ofskill in the art. In such embodiments, determining (422) by the alertanalyzer in dependence upon alert analysis rules (222) and the alerts inthe alert pool whether to suppress any alerts may be carried out bycomparing the attributes of the alerts in the alerts pool to the alertanalysis rules and identifying as a result of the comparison one or morealerts for suppression according to the event analysis rules.

The method of FIG. 6 includes transmitting (420) the unsuppressed alertsto one or more components of the distributed processing system.Transmitting (420) the unsuppressed alerts to one or more components ofthe distributed processing system may be carried out by sending amessage containing the alert to one or more components of thedistributed processing system. In many cases, an alert may be sent as amessage to a systems administrator advising the systems administrator ofone or more occurrences within the distributed processing system.

As mentioned above, alert analysis rules may select additional alerts orsuppress alerts in dependence upon events. In such embodiments,determining whether to suppress any alerts includes selecting events anddetermining whether to suppress any alerts in dependence upon theselected events. The method of FIG. 6 therefore also includesidentifying (426) by the alert analyzer in dependence upon alertanalysis rules (222), the alerts in the alert pool (224), and anyselected events one or more additional alerts and in the method of FIG.6, transmitting (428) the unsuppressed alerts also includes transmitting(430) any additional alerts to one or more components of the distributedprocessing system.

As mentioned above, relevant alert delivery according to embodiments ofthe present invention includes the administration of one or more poolsof incidents such as events, alerts or other incidents as will occur tothose of skill in the art. For further explanation, FIG. 7 sets forth aflow chart illustrating an exemplary method of administering incidentpools for event and alert analysis in a distributed processing systemaccording to embodiments of the present invention. The method of FIG. 7includes receiving (702), by an incident analyzer from an incident queuea plurality of incidents (704) from one or more components of thedistributed processing system. The incidents of FIG. 7 may beimplemented as events and the pool of incidents may be implemented as anevents pool. The incidents of FIG. 7 also may be implemented as alertsand the pool of incidents may be implemented as an alerts pool.

Receiving (702), by an incident analyzer from an incident queue aplurality of incidents (704) from one or more components of thedistributed processing system may be carried out by receiving eventsfrom an events queue by an events analyzer as discussed above. Suchevents may be sent from one or more components of the distributedprocessing system. Receiving (702), by an incident analyzer from anincident queue a plurality of incidents (704) from one or morecomponents of the distributed processing system also may be carried outby receiving alerts from an alerts queue by an alerts analyzer asdiscussed above. Such alerts may be produced by an events analyzer asdiscussed above.

The method of FIG. 7 includes creating (705), by the incident analyzer,a pool (708) of incidents, the pool having a predetermined initialperiod of time. Creating (705), by the incident analyzer, a pool (708)of incidents may be carried out by creating a data structure forinclusion of events or alerts and establishing for that data structurean initial period of time to include within that data structure events,alerts or other incidents as will occur to those of skill in the art.

The method of FIG. 7 also includes assigning (710), by the incidentanalyzer, each received incident to the pool. Assigning (710), by theincident analyzer, each received incident to the pool may be carried outin dependence upon attributes of the incident. Assigning (710) eachreceived incident to the pool for example may be carried out byselecting events, alerts, or other incidents in dependence upon suchattributes as the component creating an event, the event type, the alerttype, the component reporting the event, the component that reported orcreated the event in whose dependence an alert was created, and so on aswill occur to those of skill in the art.

The method of FIG. 7 also includes assigning (712), by the incidentanalyzer, to each incident a predetermined minimum time (714) forinclusion in a pool (708). The predetermined minimum time (714) forinclusion in the pool is typically shorter than the initial period oftime for the pool such that the first incident that arrives is includedin the pool of incidents when that pool is closed. The predeterminedminimum time may vary according to attributes of the incident as willoccur to those of skill in the art. Allowing for minimum times that varywith the attributes of the incident provides flexibly in administeringpools according to embodiments of the present invention in various formsof distributed processing systems.

The method of FIG. 7 includes extending (716), by the incident analyzer,for one or more of the incidents the predetermined initial period oftime of the pool (708) by a particular period of time assigned to theincident. The particular period of time extended to the predeterminedinitial period of time may depend upon attributes of the incident suchas the type of the incident, an identification of the incident, thecontext in which the incident is created or transmitted, and others aswill occur to those of skill in the art.

The method of FIG. 7 also includes determining (718), by the incidentanalyzer, whether conditions have been met to close the pool (708).Determining (718), by the incident analyzer, whether conditions havebeen met to close the pool (708) may be carried out by determiningwhether a maximum time for the pool has elapsed. The maximum timeestablished for the pool is typically dependent upon factors of thedistributed processing system such as the size of the distributedprocessing system, the components of the distributed processing system,the computational requirements of the distributed processing system andothers as will occur to those of skill in the art. Typically, themaximum time is established to be long enough to allow enough incidentsto accumulate into the pool for meaningful event or alert analysis butshort enough such that that analysis is not overwhelmed by the sheernumber of incidents and to allow for alerts to be reported in a timelymanner.

Determining (718), by the incident analyzer, whether conditions havebeen met to close the pool (708) may be carried out by determining thatthe predetermined initial period of time and any extensions of timeassociated with any incidents assigned to the pool have elapsed beforeanother incident is assigned to the pool. In such embodiments, thepredetermined initial period of time and all extensions of time to thepool have typically elapsed prior to a maximum time for the pool therebyclosing the pool prior to the maximum time for the pool.

If conditions have been met to close the pool, the method of FIG. 7 alsoincludes determining (720) for each incident in the pool whether theincident has been in the pool for its predetermined minimum time (714)for inclusion in a pool (708). Determining (720) for each incident inthe pool whether the incident has been in the pool for its predeterminedminimum time (714) for inclusion in a pool (708) may be carried out bycomparing the length of time each incident has resided in the pool withits minimum time for inclusion in the pool.

If the incident has been in the pool for us predetermined minimum time,the method of FIG. 7 includes including (722) the incident in the closedpool. Including (722) the incident in the closed pool is carried out bymaintaining the incident in the closed pool for event analysis, alertanalysis, or other incident analysis as will occur to those of skill inthe art.

If the incident has not been in the pool for its predetermined minimumtime, the method of FIG. 7 includes evicting (724) the incident from theclosed pool and including the incident in a next pool. Evicting (724)the incident from the closed pool and including the incident in a nextpool may be carried out by creating a next pool and moving the incidentinto the next pool. Creating a next pool and moving the incident intothe next pool may include establishing a predetermined time for the nextpool that is at least as long as the minimum time for the event moved tothat next pool.

For further explanation, FIG. 8 sets forth a flow chart illustrating anadditional method of administering incident pools for event and alertanalysis in a distributed processing system according to embodiments ofthe present invention. The method of FIG. 8 is similar to the method ofFIG. 7 in that the method of FIG. 8 includes receiving (702), by anincident analyzer from an incident queue a plurality of incidents (704)from one or more components of the distributed processing system;creating (706), by the incident analyzer, a pool of incidents (708), thepool having a predetermined initial period of time; assigning (710), bythe incident analyzer, each received incident to the pool; assigning(712), by the incident analyzer, to each incident a predeterminedminimum time for inclusion in a pool; extending (716), by the incidentanalyzer, for one or more of the incidents the predetermined initialperiod of time of the pool (708) by a particular period of time assignedto the incident; determining (718), by the incident analyzer, whetherconditions have been met to close the pool; and if conditions have beenmet to close the pool determining (720) for each incident in the poolwhether the incident has been in the pool for its predetermined minimumtime for inclusion in a pool; and if the incident has been in the poolfor us predetermined minimum time, including (722) the incident in theclosed pool; and if the incident has not been in the pool for itspredetermined minimum time, evicting (724) the incident from the closedpool and including the incident in a next pool.

The method of FIG. 8 differs from the method of FIG. 7 in that in themethod of FIG. 8 one or more of the incidents (704) are events and themethod of FIG. 8 also includes identifying (716) one or more alerts independence upon one or more events assigned to the pool (708).Identifying (716) one or more alerts in dependence upon one or moreevents assigned to the pool (708) may be carried out by identifyingalerts in dependence upon event arrival rules as discussed above,identifying additional alerts in dependence upon pool closure rules asdiscussed above, and in other ways as will occur to those skill in theart.

In the method of FIG. 8 evicting (724) the incident and including theincident in a next pool also includes evicting (802) at least one event,including (804) the evicted event in the next pool, and associating(806) any alerts identified in dependence upon the evicted event withthe closed pool. Evicting (724) the incident and including the incidentin a next pool also includes evicting (802) at least one event,including (804) the evicted event in the next pool may be carried out bycreating a next pool and moving the incident into the next pool.

Associating (806) any alerts identified in dependence upon the evictedevent with the closed pool may be carried out by maintaining any alertsidentified in dependence upon event arrival rules with the closed poolfor alert analysis by an alert analyzer. In such embodiments, while theevent is evicted and included in the next pool any alerts identified independence upon that event are maintained for alert analysis with theclosed pool.

For further explanation, FIG. 9 sets forth a flow chart illustrating anadditional method of administering incident pools for event and alertanalysis in a distributed processing system according to embodiments ofthe present invention. The method of FIG. 9 is similar to the methods ofFIGS. 7 and 8 in that the method of FIG. 9 includes receiving (702), byan incident analyzer from an incident queue a plurality of incidents(704) from one or more components of the distributed processing system;creating (706), by the incident analyzer, a pool of incidents (708), thepool having a predetermined initial period of time; assigning (710), bythe incident analyzer, each received incident to the pool; assigning(712), by the incident analyzer, to each incident a predeterminedminimum time for inclusion in a pool; extending (716), by the incidentanalyzer, for one or more of the incidents the predetermined initialperiod of time of the pool (708) by a particular period of time assignedto the incident; determining (718), by the incident analyzer, whetherconditions have been met to close the pool; and if conditions have beenmet to close the pool determining (720) for each incident in the poolwhether the incident has been in the pool for its predetermined minimumtime for inclusion in a pool; and if the incident has been in the poolfor us predetermined minimum time, including (722) the incident in theclosed pool; and if the incident has not been in the pool for itspredetermined minimum time, evicting (724) the incident from the closedpool and including the incident in a next pool.

The method of FIG. 9 differs from the method of FIG. 7 in that in themethod of FIG. 9 one or more of the incidents (704) are events (202) andthe method of FIG. 9 also includes identifying (716) one or more alertsin dependence upon one or more events assigned to the pool (708).Identifying (716) one or more alerts in dependence upon one or moreevents assigned to the pool (708) may be carried out by identifyingalerts in dependence upon event arrival rules as discussed above,identifying additional alerts in dependence upon pool closure rules asdiscussed above and in other ways as will occur to those skill in theart.

In the method of FIG. 9, evicting (724) the incident and including theincident in a next pool also includes evicting (902) at least one event,including (904) evicted event in the next pool, associating (906) anyalerts identified in dependence upon the evicted event with the nextpool. Evicting (724) the incident and including the incident in a nextpool also includes evicting (902) at least one event, including (904)the evicted event in the next pool may be carried out by creating a nextpool and moving the incident into the next pool.

Associating (906) any alerts identified in dependence upon the evictedevent with the next pool may be carried out by disassociating any alertsidentified in dependence upon event arrival rules with the closed poolfor alert analysis by an alert analyzer and associating those alertswith the next pool. In such embodiments, the event is evicted andincluded in the next pool and any alerts identified in dependence uponthat event are also associated with the next pool for alert analysiswith the next pool.

In some embodiments alerts may be identified in dependence upon otheralerts. In such embodiments, therefore, associating any alertsidentified in dependence upon the evicted event with the next poolaccording to the method of FIG. 9 also may be carried out byinvalidating any alerts or events in the closed pool dependent upon theidentified alerts now associated with the next pool. That is, alertsidentified in dependence upon alerts now associated with the next poolare invalidated in the closed pool. Invalidating such alerts removesdependencies in the closed pool on alerts now associated with the nextpool.

As mentioned above, events are often assigned to an events poolaccording to their logged time. That is, events are typically insertedinto the events pool in the order that they are received in the eventqueue. However, during events storms it is often preferable toadminister events in the events pool in dependence upon their occurredtime rather than their logged time. For further explanation, therefore,FIG. 10 sets forth a flow chart illustrating an example method ofdynamic administration of event pools for relevant event and alertanalysis during event storms according to embodiments of the presentinvention. An events storm, as that term is used here, is the result ofan occurrence in the distributed processing system that causes anoverwhelming number of events reported by components of the distributedprocessing system. Such occurrences that may cause an events storm mayinclude the loss of an entire circuit providing power to many componentsof the data processing system, catastrophic failure of a number ofcomponents of the distributed processing system and others as will occurto those of skill in the art. Events storms are often calledreliability, availability, and serviceability ‘RAS’ storms.

The method of FIG. 10 includes receiving (902), by an events analyzerfrom an events queue, a plurality of events (202) from one or morecomponents of the distributed processing system, each event including anoccurred time (904) and a logged time (906). As mentioned above, eachevent (202) typically includes an occurred time (904) and a logged time(906). The occurred time (904) represents the time the event wasgenerated as a result of the occurrence that cause the event. The loggedtime (906) is the time in which the event is included in the eventsqueue. During events storms, it is often preferable to administer eventspools in dependence upon the occurred time rather that the logged timeof an event.

The method of FIG. 10 also includes creating (908) by the event analyzeran events pool (212). Creating (908) by the event analyzer an eventspool (212) may be carried out by creating a data structure for inclusionof events and establishing for that data structure an initial period oftime to include within that data structure events according to eithertheir logged time or their occurred time.

The method of FIG. 10 also includes determining (910) whether an arrivalrate (912) of the events (202) from the components of the distributedprocessing system is greater than a predetermined threshold (914).Determining (910) whether an arrival rate (912) of the events (202) fromthe components of the distributed processing system is greater than apredetermined threshold (914) may be carried out by counting the numberof events received by the events analyzer over a predetermined period oftime and comparing that ratio to the predetermined threshold (914).

If the arrival rate is not greater than the predetermined threshold, themethod of FIG. 10 includes assigning (918), by the events analyzer, aplurality of events (202) to the events pool (212) in dependence upontheir logged time (904). Assigning (918), by the events analyzer, aplurality of events (202) to the events pool (212) in dependence upontheir logged time (904) may be carried out by selecting events having alogged time after the start of a pool and before the close of a pool.Such events may also be selected and assigned to a particular pool independence upon other attributes such as the component creating theevent, the event type, the component reporting the event, and so on aswill occur to those of skill in the art.

If the arrival rate is greater than the predetermined threshold, themethod of FIG. 10 includes assigning (916), by the events analyzer, aplurality of events (202) to the events pool (212) in dependence upontheir occurred time (906). Assigning (916), by the events analyzer, aplurality of events (202) to the events pool (212) in dependence upontheir occurred time (906) includes selecting events having an occurredtime after the start of a pool and before the close of a pool. Suchevents may also be selected and assigned to a particular pool independence upon other attributes such as the component creating theevent, the event type, the component reporting the event, and so on aswill occur to those of skill in the art.

Assigning (916), by the events analyzer, a plurality of events (202) tothe events pool in dependence upon their occurred time (906) may becarried out by determining whether the occurred time of the event isafter a start time of the pool and before a close time of the pool. Thestart time of the pool may be implemented as the time a first event isassigned to the pool. That is, typically a new pool may be created uponthe first event received after a last pool is closed.

The close time of the pool may be implemented as a predetermined initialperiod of time after the start time and any extensions of timeassociated with any events assigned to the pool. Alternatively, theclose time of the pool may be a maximum duration of time after the starttime.

In the method of FIG. 10, after a period of time a new arrival rate(920) may be calculated and may be beneath the predetermined threshold(914). In such situations, method of FIG. 10 includes determining (910)that a new arrival rate (920) of the events (202) from the components ofthe distributed processing system is less than the predeterminedthreshold (914) and assigning (918), by the events analyzer, a pluralityof events (202) to the events pool in dependence upon their logged time(904). Assigning (918), by the events analyzer, a plurality of events(202) to the events pool (212) in dependence upon their logged time(904) may be carried out by selecting events having a logged time afterthe start of a pool and before the close of a pool. Such events may alsobe selected and assigned to a particular pool in dependence upon otherattributes such as the component creating the event, the event type, thecomponent reporting the event, and so on as will occur to those of skillin the art.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

It will be understood from the foregoing description that modificationsand changes may be made in various embodiments of the present inventionwithout departing from its true spirit. The descriptions in thisspecification are for purposes of illustration only and are not to beconstrued in a limiting sense. The scope of the present invention islimited only by the language of the following claims.

1. A method of dynamic administration of event pools for relevant eventand alert analysis during event storms, the method comprising:receiving, by an events analyzer, a plurality of events from one or morecomponents of the distributed processing system, each event including anoccurred time and a logged time, wherein the occurred time is a time theevent was generated as a result of an occurrence that cause the event,and wherein the logged time is a time the event is included in theevents queue; creating, by the event analyzer executing by a computerprocessor, an events pool; determining whether an arrival rate of theevents from the components of the distributed processing system isgreater than a predetermined threshold; if the arrival rate is greaterthan the predetermined threshold, assigning, by the events analyzer, aplurality of events to the events pool with an occurred time after astart time of the pool and before a close time of the pool; and if thearrival rate is not greater than the predetermined threshold, assigning,by the events analyzer, a plurality of events to the events pool with alogged time after a start time of the pool and before a close time ofthe pool.
 2. (canceled)
 3. The method of claim 1 wherein the start timeof the pool comprises the time a first event is assigned to the pool. 4.The method of claim 1 wherein the close time of the pool comprises apredetermined initial period of time after the start time and anyextensions of time associated with any events assigned to the pool. 5.The method of claim 1 wherein the close time of the pool comprises amaximum duration of time after the start time.
 6. The method of claim 1further comprising: determining that a new arrival rate of the events ina database is less than the predetermined threshold; and assigning, bythe events analyzer, a plurality of events to the events pool independence upon their logged time. 7-20. (canceled)